Expand e2e testing#151
Conversation
|
Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (13)
📝 WalkthroughWalkthroughThe PR adds self-managed CDP E2E execution, shared-instance recovery helpers, feature tests, tier-specific CI artifacts, and documentation. It also changes project modal discard guards to include pending edits through a renamed ChangesUnsaved-work modal guarding
E2E test infrastructure and feature coverage
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
9e3e8d2 to
66fc3fc
Compare
alex-rawlings-yyc
left a comment
There was a problem hiding this comment.
@alex-rawlings-yyc resolved 4 discussions.
Reviewable status: 0 of 56 files reviewed, all discussions resolved (waiting on alex-rawlings-yyc).
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
ca70ddd to
e9c1bed
Compare
|
When I pull this branch and run When I run |
|
Whoops. I'll have another look. I assume we want |
alex-rawlings-yyc
left a comment
There was a problem hiding this comment.
@alex-rawlings-yyc resolved 1 discussion.
Reviewable status: 0 of 70 files reviewed, all discussions resolved (waiting on alex-rawlings-yyc).
imnasnainaec
left a comment
There was a problem hiding this comment.
@imnasnainaec partially reviewed 53 files and made 1 comment.
Reviewable status: 50 of 73 files reviewed, 1 unresolved discussion (waiting on alex-rawlings-yyc).
.github/workflows/test.yml line 114 at r9 (raw file):
if ($LASTEXITCODE -eq 0) { exit 0 } Write-Host "npm ci failed (exit $LASTEXITCODE); verifying cache before retry" npm cache verify
Claude suggestion from a Devin nit:
worth a one-liner after npm cache verify (e.g., if ($LASTEXITCODE -ne 0) { Write-Host "npm cache verify failed (exit $LASTEXITCODE)" } for clearer diagnostics)
Exclude the "Open Interlinearizer" project-picker tab from the four broad .dock-tab matches (ensureInterlinearizerOpenOnWeb, isDraftDirty, closeInterlinearizerTab, rescueDraftToNewProject) so they target the real WebView tab. Also anchor the active-project check in ensureE2eProjectActive to match E2E_PROJECT_NAME exactly instead of via unanchored includes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The e2e feature helpers couldn't reliably pass on the shared CDP
instance:
- ensureE2eProjectActive matched the project entry by the button's
accessible
name, but the modal renders name/badge/language spans with no
separating
whitespace, so the anchored regex never matched — every mutating test
fell
into the create branch and re-created an existing project. Match the
name
element by exact text instead (also fixes the "E2E Test Project 2"
overmatch).
- The create branch hung on a dirty draft (Create defers behind the
discard
confirm); dismiss it when dirty.
- closeInterlinearizerTab used hover()+click(), which times out when the
tab
overflows off-viewport on CI; use dispatchEvent('click') like
paranext-core.
- ensureInterlinearizerOpenOnWeb settles the dock before its isVisible()
branch.
- Extract interlinearizerTabLocator/escapeRegExp helpers.
The tab's unsaved marker used dirty||pendingEdits but ProjectModals
gated its
discard confirmation on committed dirty only, so opening/creating a
project
mid-typing silently discarded the uncommitted gloss. Rename the prop to
hasUnsavedWork and feed it the combined state so the guard and marker
agree.
Document the broadened discard-on-swap behavior in user-questions.md.
globalSetup now launches and tears down its own Platform.Bible instance (reusing a warm start:cdp when the CDP port is in use), and .or() tab waits get .first() so a docked Interlinearizer + WEB tab no longer trip Playwright strict mode.
Kill the CDP app's whole process tree (taskkill /T /F on Windows, POSIX process-group SIGKILL elsewhere) so no orphans hold port 8876, and stream the launched app's output to a log instead of discarding it, so a startup crash shows its cause rather than an opaque WebSocket-port timeout.
* Suggested changes on expand-e2e-testing * Fix Linux CI
Self-heal any modal a prior failed test left mounted (dismissLeftoverModals in ensureInterlinearizerOpenOnWeb) so its full-viewport overlay can't intercept every downstream test's clicks, force-click through the wipe modal's settle race, and enable CI retries now that each attempt starts clean. Also race the CDP-port wait against the early-exit sentinel so a late startup crash fails fast instead of timing out.
Disable electron-debug's auto-opened DevTools (ELECTRON_IS_DEV=0) and pin the window to 1280x960 so dock panels and modals are not clipped; gate readiness on dock-tab titles resolving (no "Unknown" tabs) in both waitForAppReady and CDP global setup so a broken instance fails fast with its startup log; upload the dotfile startup log (include-hidden-files) and force X11 on Linux so headless xvfb runs work on Wayland machines.
The e2e job's full core install intermittently fails on windows-latest with EEXIST in the shared npm cache; retry up to 3x with cache verify between attempts. Linux keeps its single-shot install.
The "Open Interlinearizer" picker is a floating dock tab that opening a project never disposes; on the shared CDP instance it accumulated until a bare .select-project-dialog read tripped strict mode and reddened every downstream test. Scope the locator to .first(), close the picker after selecting a project, and self-heal leftover pickers in the open precondition. Also log npm cache verify failures during the Windows ci retry.
navigateToScriptureRef drove the platform toolbar's book-chapter control, which is permanently disabled in simple interface mode with no main-editor project (the CDP instance's state) — so trigger.click() retried for the full test timeout, hanging every navigating feature test for 6 minutes. Wait a bounded time for the control to enable and skip navigation if it never does; the view already shows its default reference and callers assert on tokens.
The dock-readiness gate failed if any tab anywhere said "Unknown", so a single leftover panel poisoned every later feature test on the shared instance. Make it accept a lenient mode (dock mounted + >=1 tab resolved) for CDP feature tests while smoke/cold-start stay strict, report a closed-page teardown distinctly, and capture the smoke app's stdout.
The readiness wait used the 120s cold-start budget on every attempt, so a dead shared instance cost ~6 min per test (120s x 3 retries) before reporting. Add CDP_FEATURE_READY_TIMEOUT (30s), passed by the feature specs, and split waitForAppAndInterlinearizerReady's timeout across both the app-ready and extension-registration waits so the short budget caps the whole check — a dead instance now fails in ~90s. Smoke tests keep the generous cold-start default.
63965e3 to
446ddcd
Compare
Wait on the settings/menu-data/theme data-provider objects in rpc.discover before the dock-tab-title check, so a slow cold start resolves against its real cause instead of an opaque "Unknown"-tab timeout. Also fast-fail the cold-start wait on the fatal theme-settle page error, turning a 120s dead wait into a fast, labeled failure that retries onto a fresh launch.
alex-rawlings-yyc
left a comment
There was a problem hiding this comment.
@alex-rawlings-yyc resolved 1 discussion.
Reviewable status: 50 of 73 files reviewed, all discussions resolved (waiting on alex-rawlings-yyc and imnasnainaec).
Cut paranext-core internals (serialization shape, magic numbers, symbol names) that go stale silently; keep the observable contract and flag the few comments that still track an upstream detail.
Also fix the smoke launcher's dual stdout/stderr pipe: use { end: false
}
so neither stream ends the shared log early, and flush it before
dumping.
Replaces the manual process-group kill in launchElectronWithExtension and teardownElectronApp with the shared killProcessTree utility, matching both global-teardown files and closing the Windows tree-kill gap.
The dock-tab and service-host waits floored their leftover budget to 1ms (Math.max(1, ...)), which clears waitForDockTabTitlesResolved's timeout<=0 guard but expires during the CDP round-trip before the first predicate evaluation — failing "tabs still Unknown" on a healthy app. Floor to 1000ms to match waitForAppReady, and correct its stale "mirrors Math.max(1, ...)" comment.
imnasnainaec
left a comment
There was a problem hiding this comment.
npm run test:e2e passes for me locally on Windows 11 with the latest paranext-core (3eb6556cfd2e37d1610c3fcf873a08ed4892f635). I just have a handful of nit-picks so far and should finish reviewing tomorrow afternoon.
@imnasnainaec reviewed 15 files and made 6 comments.
Reviewable status: 65 of 73 files reviewed, all discussions resolved (waiting on alex-rawlings-yyc).
.github/workflows/test.yml line 102 at r12 (raw file):
working-directory: paranext-core shell: pwsh # Same install as Linux (no --omit=optional: lightningcss native binaries are optional deps
⛏️ Unnecessary parenthetical duplicated from above.
e2e-tests/playwright.config.ts line 23 at r12 (raw file):
retries: process.env.CI ? 2 : 1, workers: 1, // Tier-specific report/output folders so `npm run test:e2e` (smoke then cdp, sequentially) doesn't
⛏️ The first sentence of this comment feels unnecessary.
e2e-tests/README.md line 12 at r12 (raw file):
Both tiers are self-launching: the CDP tier's `globalSetup` launches its own Platform.Bible instance (with `--remote-debugging-port=9223`) in an isolated user-data dir and tears it down afterward, so `npm run test:e2e:cdp` needs no manual `npm run start:cdp` first. To iterate against a warm instance instead, run `npm run start:cdp` in one terminal, then run the CDP config directly with `npx playwright test --config e2e-tests/playwright-cdp.config.ts`: the setup detects the in-use CDP port, reuses that instance, and leaves it running. In CI (`.github/workflows/test.yml`, `e2e` job) the full suite runs on both Linux and Windows. Each tier writes its Playwright HTML report to its own subfolder (`playwright-report/smoke`, `playwright-report/cdp`) so a combined run keeps both.
⛏️ The sentence about subfolders feels superfluous here in the README.
e2e-tests/README.md line 39 at r12 (raw file):
- **The instance is only ever used with the WEB project.** This is an operating assumption, not something tests verify: `ensureInterlinearizerOpenOnWeb()` trusts an existing Interlinearizer tab and only picks WEB when opening fresh. Don't point `start:cdp` at other projects. - **Mutating tests operate on the dedicated "E2E Test Project", never on a developer's own projects.** `ensureE2eProjectActive()` opens it (creating it on first use) at the start of each mutating test. Because the draft is the single per-source working buffer, replacing it could destroy unsaved developer work — so when the draft is dirty and the active project is _not_ the e2e project, the helper first saves the draft into a new `e2e-rescued-work-<timestamp>` project. Rescue projects are backups, not junk: delete them manually once recovered. Dirty state left while the e2e project is active is treated as leftover test data and discarded. - **Self-establish every precondition.** Each mutating test starts with `ensureInterlinearizerOpenOnWeb()` → `ensureE2eProjectActive()` → `navigateToScriptureRef()` (the scroll-group reference could be anywhere) → `wipeDraft()`.
⛏️ If we're documenting that every test of a given class begins with a chain of 4 functions, perhaps we should wrap those in a single function.
e2e-tests/README.md line 40 at r12 (raw file):
- **Mutating tests operate on the dedicated "E2E Test Project", never on a developer's own projects.** `ensureE2eProjectActive()` opens it (creating it on first use) at the start of each mutating test. Because the draft is the single per-source working buffer, replacing it could destroy unsaved developer work — so when the draft is dirty and the active project is _not_ the e2e project, the helper first saves the draft into a new `e2e-rescued-work-<timestamp>` project. Rescue projects are backups, not junk: delete them manually once recovered. Dirty state left while the e2e project is active is treated as leftover test data and discarded. - **Self-establish every precondition.** Each mutating test starts with `ensureInterlinearizerOpenOnWeb()` → `ensureE2eProjectActive()` → `navigateToScriptureRef()` (the scroll-group reference could be anywhere) → `wipeDraft()`. - **Reset at the start, tidy at the end.** The start-of-test sequence is what guarantees correctness — it self-heals whatever a failed run left behind. Part of that self-heal is automatic: `ensureInterlinearizerOpenOnWeb()` calls `dismissLeftoverModals()`, which cancels any modal a timed-out prior test left mounted — a project modal's overlay is `fixed inset-0 z-50` and would otherwise intercept every click in the tests that follow, turning one real failure into a cascade. This is also why the CDP config can safely `retries` in CI: the retry lands on a self-healed instance, not on the overlay the failed attempt left up. Mutating tests additionally end with `ensureE2eProjectActive(page, { rescueDirtyDraft: false })` to discard their own leftovers; this is a courtesy so the next run doesn't misread test junk as rescuable developer work, not something correctness depends on.
⛏️ The full details of the reset-at-start and tidy-at-end should be in the functions' JSDocs. A more general description would be more fitting here.
Drop duplicated CI-comment detail, redundant README lines, and document why the mutating-test setup stays unwrapped.
alex-rawlings-yyc
left a comment
There was a problem hiding this comment.
@alex-rawlings-yyc made 5 comments.
Reviewable status: 65 of 73 files reviewed, all discussions resolved (waiting on alex-rawlings-yyc).
.github/workflows/test.yml line 102 at r12 (raw file):
Previously, imnasnainaec (D. Ror.) wrote…
⛏️ Unnecessary parenthetical duplicated from above.
Done. Trimmed the parenthetical. The "Same install as Linux" opener already points back to the Linux comment above, so the --omit=optional/lightningcss detail only lives in one place now.
e2e-tests/playwright.config.ts line 23 at r12 (raw file):
Previously, imnasnainaec (D. Ror.) wrote…
⛏️ The first sentence of this comment feels unnecessary.
Done. Dropped the "smoke then cdp, sequentially" lead-in. The tier-specific report/output folders rationale reads fine on its own.
e2e-tests/README.md line 12 at r12 (raw file):
Previously, imnasnainaec (D. Ror.) wrote…
⛏️ The sentence about subfolders feels superfluous here in the README.
Done. Removed the subfolder sentence from the CI paragraph. That detail now lives only in the playwright.config.ts comment where the folders are actually configured.
e2e-tests/README.md line 39 at r12 (raw file):
Previously, imnasnainaec (D. Ror.) wrote…
⛏️ If we're documenting that every test of a given class begins with a chain of 4 functions, perhaps we should wrap those in a single function.
I'd rather keep these as separate calls than wrap them. The chain isn't actually uniform: the draft-persistence reopen (draft-persistence.spec.ts:54-55) deliberately runs only ensureInterlinearizerOpenOnWeb() + navigateToScriptureRef(), skipping ensureE2eProjectActive() and wipeDraft() — re-establishing the project or wiping the draft would destroy the exact state that test is verifying. A single wrapper would hide that meaningful subset-selection. I've reworded the bullet to say the steps compose as needed and to call out the reopen case, so the "why not one function" is documented. Happy to reconsider if you still want a wrapper for the common all-four setup.
e2e-tests/README.md line 40 at r12 (raw file):
Previously, imnasnainaec (D. Ror.) wrote…
⛏️ The full details of the reset-at-start and tidy-at-end should be in the functions' JSDocs. A more general description would be more fitting here.
Agreed. Condensed the bullet to a general description that points at the JSDocs on ensureInterlinearizerOpenOnWeb() and ensureE2eProjectActive(). Those already carry the full reset-at-start / tidy-at-end mechanism (the modal/picker self-heal and the rescueDirtyDraft semantics), so nothing was lost in the trim.
imnasnainaec
left a comment
There was a problem hiding this comment.
@imnasnainaec reviewed 4 files and all commit messages, and made 2 comments.
Reviewable status: 69 of 73 files reviewed, all discussions resolved (waiting on alex-rawlings-yyc).
e2e-tests/global-teardown-cdp.ts line 23 at r13 (raw file):
// need it fully dead before removing its user-data dir below. let appKilled = false; if (fs.existsSync(CDP_PID_FILE)) {
⛏️ Since the content of this if block is nearly identical to the corresponding block in e2e-tests/global-teardown.ts, could it be extracted as an exported function there that can be imported and used here?
e2e-tests/tests/features/draft-persistence.spec.ts line 25 at r13 (raw file):
// (test.slow() above triples only the overall test timeout for the two open-cycles below, not // this startup readiness check.) await waitForAppAndInterlinearizerReady(mainPage, {
⛏️ Since the same options (and similar justifying comments) are present in every features/ use of waitForAppAndInterlinearizerReady, can we consider simplifying it to a single bool param (e.g., lenient or cdp)? Then all these comments and the docs on the options type could all be absorbed into the docs for the single param.
Extract killProcessFromPidFile so global-teardown and its CDP variant
share one PID-file kill block. Replace the repeated { strict, timeout }
readiness options with a single { cdp: true } flag on
waitForAppAndInterlinearizerReady, moving the justifying comment onto
the option's docs.
Trim the readiness/teardown/launch commentary across the e2e suite to its load-bearing details; no behavior changes.
alex-rawlings-yyc
left a comment
There was a problem hiding this comment.
@imnasnainaec addressed nits; both worthwhile adjustments.
@alex-rawlings-yyc made 1 comment.
Reviewable status: 69 of 73 files reviewed, all discussions resolved (waiting on alex-rawlings-yyc).
This change is
Summary by CodeRabbit